DOCOHENT EBSOBE 



ED U2 582 



TH 006 ai9 



ADTHOR 
TITLE 

INSTITUTION 

REPORT NO 
EDB DATE 
NOTE 

EDRS PRICE 
DESCRIPTORS 



Applegate^ Neil J. 

Test and Evaluation of Criterion Testing in the 

Format of Discrete Motion Pictures, 

Naval Education and Training Command^ Pensacola^ 

Fla. 

'CNErS-E-il-76 
Jun 76 
32p, 

f1F-$0.83 HC-$2.06 Plus Postage, 

^Audiovisual Programs; Comparative Testing; Control 
Groups ; =^Cr iter ion Refer enced Tests ; ^Enlisted 
Personnel; *Eva luation Methods ; Experimental Groups ; 
Instructional Design; *I nstructional Films ; Post 
Testing; Pretesting; Single Concept Films; *Testing 
Programs; Training Techniques 



ABSTRACT 

Tvo different control groups were used to evaluate 
the use of pre- and post-test examination questions presented in the 
format of a Navy training film. The control group was given a 
paper-and-pencil pre- and post- test while the experimental group 
received the pre- and post-test questions within the context and 
format of the film. While no significant differences were revealed 
between the two groups or the two methods of testing, it was found 
that learning gains attributed to the use of the training film were 
quite outstanding when comparing student pre- and post-test 
differences. (MV) 
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1,0 guirina ry > Critorion t^stinq in the format of a Discr(^te 
Motion Picture — Oxycen ^reathinc^ Apparatus^ OBA Film MM 31169 — 
was tested an^^ evaluated. Two different recruit arouos, 
Control T and Fxperimental II were used to test and evaluate 
the use of pre- an^ tH)j^t-+.i st examination questions presented 
in the frtTvit of a Nav^ ^r.iininq film. The Control crroup and 
the Experimental group were sub-divided into three sections. 
Control Group T included sections A, C, and the Experimental 
Group II included sections E, F. Each section in each qroup 
was scheduled individually to participate in the study/evaluation 
procedure. The Control crrouo members v/ere given a printed 
(paper . nd pencil) pre-- and post-test while the Experimental 
group members were administered the same pre- and post-test 
questions by means of motion picture film. 

The results of the study revealed no significant differ-^ 
ences in test scores between the two groups and/or the two 
methods of testing (paper vs film) which can be accounted for. 
A serendipitous finding, however, revealed that learnina gains, 
made by using a structured instructional film, were quite 
outstanding when comparing student pre- and post-test differ- 
ences. Recommendations suggest that the Navy can judiciously 
prepare film type trainina packages for various informal 
training environments (OBT, GMT) and expect practically the 
same results in group paced training situations without the 
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facilities airl n.mpnwe^r roriuired in tho formal instructional 
situation. The economic data resulting from this study would 
tend to discourage a maior shift to total instructional test- 
ing by film in the Navy Discrete Motion Picture MN Film series. 
2.0 Acknowledgements . This CNET SUPPORT task could not have 
been completed without the cooperation of the Recruit Training 
Command at NTC San Diego, CA and especially LCDR Gary L. 
JOHNSON, (since transferred) . Special appreciation is also 
extended to NAVEDTRASUPPCENPAC and the Training Analysis and 
Evaluation Department personnel including Mr. Leonard Auguiar, 
Dr. J. Scott Newcomb ^ and Ms. Sandra Drummer who tested the 
subjects, organized the data, and provided the statistical 
summary. The services of Ms. Helen Bennett and Messrs. Lloyd 
Jordan, William K. Treynor, Art Crevensteh, and Robert Myers 
of the NAVTRAEQUIPCFN, Orlando, FL are grai'-.fully acknowleged. 
Their efforts produced the required art work and the instruc- 
tional users Quide for the experimental phase of this study. 
Special note is offered to Messrs. Robert Walcher, Gene 
Doubleday, and Zane G. Crockett of NAVINSTECHDEVCEN , San Diego, 
CA for their technical assistance and expertise in modifying 
the experimental film. Appreciation is also extended to Miss 
Teresa Hindman for her careful preparation of the final 
document . 

3.0 Purpose. This report presents the results of CNET SUPPORT 
Plan # 157 A 14Z IC 045 entitled A Proposed Plan To Incorporate 
Criterion Testing In The Format Of Discrete Motion Pictures . 
The purpose of this plan was to test, evaluate, and if success- 
©'"il, legislate the specifications to implement the idea of 
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incorporatinc^ rritorion tostinc; in the format of all discrete 
motion pictures nroducetl for Navv trainino • 
4.0 Backcrroun 1 . This ta^^-: oriainatod as a result of CNET 
SUPPORT sta^f units wr^rkina tn rlevelop specifications fox 
use in thf^ production nf :Javal Traininq Discrete Motion 
Pictures, e.a- N^vy Filir, 11 36^, O peration and Use of 
Emergency ^'^xyaen Breathing Apparatus (OPA) ■ To this point, no 
one his reporter hard data relative to the effectiveness and 
econony of ^^-^stina stuients bv usina filmed questions versus 
the printed pacro. Until now, questions such as those v/hich 
follow have remained ^;nansv:ered . 

* I*7ould there be anv significant difference in student 
scores by testina students through the medium o^ film projected 
test questions compared tn using the printed questionnaire? 

* Would there be a cns^ avoidance by eliminating printed 
pre- and post-test questionnaires? 

* Would there be any distinct advantage in terms of 
production costs when comparing film to the printed page, etc.? 
To this end a sppcial edition of Maw film MN~11369 was developed 
and a plan was devised to determine the answers to these 
questions. Film revision was performed in-house and included 
only the addition of the OBA training film criterion test 
questions used in a previous study which measured the Training 
Effectiveness of Films Developed Using Systems Approach to Train- 
ing Principles (CNETS Report 4-75 refers). The film scenarios 



in the previous trainina Trackage had been desianed around an 
established set of specific behavioral objectives (SROs) , 
thus, the instructional ^Toals of the film were structureH 
and the test measurpvS wero developed and printed accordinqly. 
The study reported here would eliminate the printed (oaper) 
questionnaire. The findings, as described in paraaraoh 6.0 
and Tables 2-7, were submitted in MA^^EDTRASUPPCENPAC letters 
N1:LMA: JSriSKDiSA 1500 Ser ^29 of 3 May 1976 and N1:JSN:LMP 
1500 Ser of 24 August 197P. 

5.0 Me thods . Information reaarding the subjects involved 
in this study aloncr with the testing procedures and the 
statistical analysis used to brincr this study to a conclusion 
are presented here. 

5.1 Sub j ects . Two different groups of subjects were used 
in this evaluation. One group was identified as Control 
Group #1 and one as Fxperim.ental Group #11. Both groups wera 
divided into three sections of at least 30 persons per section. 
In reality, a total of 192 persons participated in this study. 
Ninety two served in the Control group and one hundred served 
with the Fxperimental aroup. Roth aroups were exposed to 

the sane pre- and post-test questions concerning the Operation 
and Use of Emergency Rreathina Apnaratus Type A-three; however, 
only the Experimental aroup v/as aiven the pre- and post-test 
by means of the modified training film itself. Table 1 presents a 
summary of the groups and individual sections used in this 
study. As the table shovjs , the two groups involved in this 
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study were recruits in basic traininq at the Naval Recruit 

Training Command, San Dieao, CA . None of the recruits involved 

in this study had had formal or informal Navy training in 

the operation or usf^ of the oxygen breathing apparatus. 

The Control group viev/e'' tLo standard Navy Training OBA film 

MN-31169 and received both the pre- and post- criterion tests 

by traditional pencil and paper means. 

5.2 Testing Procedures . As mentioned above in paragraph 5.1, 
the OBA film was technically modified in order to incorporate 
the pre- and post- criterion test questions 'in the format of 
the film. '^he film scenarios depictinq the desired SBO's to 
be learned were not changed. Both study groups observed the 
same film content and responded to the same test questions in 
both the pre- and post-tests. The test questions had been 
previously (CNETS Report 4-75 refers) designed to measure 
individual learning based upon pre-established SBO's as developed 
through the use of the Systems Approach to Training .(SAT) 
procedures . 

In keeping with SAT principles, pre- and post-testing 
questionnaires and test administration procedures were built 
into the film. The pre-test, which was administered to the 
Control group prior to viewing the film content, measured the 
entrance level knowledge of the subjects. The post-test was 
administered followina the film in order to measure individual 
SBO achievement by each of the subjects. A list of the test 
items for the OBA film MN 31169 is Presented in Appendix A. 
The multiple answer test items were recorded by the subjects 
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on a standard optical scan test answer sheet number SM-0106-017~ 
0400. Prior to the showing of the OBA experimental film each 
study section was given explicit instructions relative to the 
use of the testing materials . In the case of Experimental 
Group II; only one optical scan test answer sheet v;as distri- 
buted prior to viewing the film. Both the pre- and post-test 
answers were recorded on the one answer sheet by utilizing the 
reverse side; however, both the pre- and post-test questions 
were read from the viewing screen rather than from the printed 
page . 

5.3 Study Procedures , The recruit subjects were randpmly 
selected and arbitrarily placed into two groups. These groups 
were further sub-divided into three sections which produced 
a total of six sections. This division is illustrated in 
Table 1. 

5.3.1 Facilities . Both phases (Control and Experimental) 
of this study were conducted in an appropriate room equipped 
with student desks, a portable movie screen, one 16mm 

sound movie projector, an appropriate supply of multiple choice 
"op scan" test answer forms and No. 2 pencils. 

5.3.2 Testing . The testing phase of this study was adminis- 
tered to each of the sections in Groups I and II on an 
individually scheduled basis. The testing plan for this study^ 
is also illustrated graphically in Table 1, 
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5.4 Statistical Analysis . Comparisons between groups were 
made using t--tests (t .05=1.98) adjusted for unequal variances. 
As a consequence of finding the pre-test Control group and 
Experimental group variances to be heterogeneous, the Cochran 
and Cox method was used to test the difference between pre-test 
means."'' Tables 2 and 3 summarize those data and present the 
t-'test results. Because Table 5 shows the post-test Control 
and Experimental group frequency distributions to be somewhat 
negatively skewed, a nonparametric .05 level test of the differ- 
ence betwoen the medians of these two grouDs was conducted. The 
results are reported in Table 6. 

5.5 Cost Analysis .- One important outcome of this study 
resulted in a comprehensive comparison of media production 
costs. The data presented in thxE study were arrived at by 
obta - .ing 1^76 cost estimates for black and white film footage 
prod^-'-^tion and by formulating the assumptions which follow: 

" Assumptions : 

Twenty five (25) prints of the OBA training film will 
be TT ified to include the required criterion test questions 
as used in this study. 

2, Each reproduction (print) of this film will be used 
one time per week for fifty (50) weeks and viewed by 15 persons 
at each showina. 



Ferguson, George A., 1966 vStatistical Analysis in Psy- 
chology and Education . 2d ed, New York: McGraw-Hill Book 



Company, pg* 171* 
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3. The life expectancy of a nrint is three years • 

4. Each paoer test v;ould rpquire two (2) sheets of 
paper at a cost of each. 

5. The same answer sheet will be used in either test 

case . 

6. The film will be retained by the user for one year. 

7. Test sheets will require replacement at least once 
per quarter. 

Analyzing media production costs to add the pre- and post-test 
to an MN training film similar to the one used for this study 
produced the costs which follovz: 

• 25 prints - $1131 • 100 Prints - $2407 

• 50 prints - $1618 • 200 Prints - $4201 

• 300 prints - $6020 

Film costs per print, excluding the lifetime maintenance cost;, 
is 1^45.24 compared to a paper cost of $18.00. 

6 . 0 Results 

6.1 Statistical Tests . Based on the data contained in Tables 
2-6 the results of the subject study are as follows: 

• At the ^05 level of significance there appears to be no 
difference in test scores botween student groups that can be 
accounted for in tho mode testing — paper or film. 

• The nonparametric .05 lev^^l test relative to the nega- 
tively skewed nost-test frequency distributions in Table 5 
supports the parametric fin^Hna reported above. 

6.2 Cost Evaluation . Althouah the mode of trusting appears 

to make no difference in student tost results^ economic analysis 

• Based upon film cost estimates provided bv Mr. William 
Trey nor, NAVTRAEOUIPCEN, Autovon 7 91-4 714, 



of relative^ nrod\i'^i- ion cns^^-^ to inc.->r^ ^ratre criterion measure 
test questions in trainina filrr fornvits tend to be nrohibitive 
and favor pa^er ly^rinted quostrionnairos) as the more cost 
efficient moth^vl ^ - bo used. 

7.0 Disrassin n * Cc:\'rot^' dita noints to ?hG 'Ea^t that there 
is no significant dif ft-. t»-*i?e in student t^st score results 
when questions are deliverec^. bv film or the printed paqe . 
Although the data in this study does not support the hypothesis 
that testing by film, would be more advantaqeous from the stand- 
point of trainina ef f ectiveness. .and te^'ting production costs, 
. a serendipi tir^us finding has been noted. Table 7 supports 
previo s findinas in CNETS Report 4-75 Training Effectiveness 
of Films Developed losing Systems Approach to Training Principles 
that post-test comparisons of the experimental and control 
groups showed that structured training? films employing the 
systems approach to training (SAT) significantly improved 
learning gains in both groups. This finding becomes significant 
to the trainina community in that both the control and experi- 
mental groups represented a heterogeneous whole who met or 
surpassed the minimum Naval recruiting standards for enlisted 
personnel. As there is no significant difference in test 
scores between crroups exposed to the group mode of learning 
and testing (film versus paper and pencil questionnaire) the 
serendipity observed above suggests that requ " ^-ed Naval 
training/learning can possibly be given to het ero'^oreous 
groups under various environmental conditions — sans the 
formal c'^assroom — and during periods o^ scheduled and unscheduled 
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inactivity, £.£. durincr holding periods between assignments on 
shore or at sea. General Military Training (GMT) and certain 
onboard trainircr (OBT) courses would be appropriate for this 
mode of instruction provided the subject film is designed 
within the parameters of SAT. Production costs notwithstanding 
serious deliberations should be given to the tyoe media and the 
mode of delivery when GMT and OBT training courses are being 
planned or revised. 

8 . 0 Conclusions . The following conclusions are made from 
the results reported above: 

8.1 Although recruits in the control aroup posted slightly 
higher test scores when post-tested with the printed question- 
naire than recruits in the experimental group who experienced 
the sane test by viewing filmed questions, there was no signi- 
ficant difference established in the statistical data to bring 
about or suagest a major modification of existing criterion 
testing procedures and/or technique. 

8.2 Pre-test knowledge of the subject in both the control 
and the experimental groups involved in this study did not 
appear to modify or influence post-test achievement by either 
group . 

8.3 Criterion test production costs for' printed questionnaires 
are appreciably less than those costs required to produce the 
same number of questions in a training film; however, the 
findings in this study suagest that initial increased produc- 
tion costs may be offset in terms of a cost benefit if the 
sane amount of learnina can take place in large group-paced 



training situations involvina required MN films and GMT, OBT 
types of courses. 

8.4 Although film costs v;ould discourage the development of 
a complete new series of MN training films which incorporate 
criterion test questionnaires, a serendipitous observation in 
this study indicates that learning achievements in both groups 
are , indeed / quite outstanding . The impact of this fact on 
future development and design of training films in the MN 
series, GMT, and OBT areas suagests the continued use of SAT 
procedures v/hen film is chosen as the instructional training 
media . 

9 . 0 Recomnendations 

9.1 The NAVEDTRACOM should continue to use printed pre- and 
post-test questionnaires as required by the Discrete Motion 
Picture MN film series; however, curricula developers should 
not limit future training film criterion test production to 
the printed page if the subject matter of the training is such 
that it can be delivered in the group paced mode under environ- 
mental conditions unlike the formal classroom. 

9.2 The findings of this study suggest that it may be both 
practical and cost effective to offer personnel shipboard 
orientation training programs related to required shipboard 
duties involving team type activities, i.e. damage control, 
f iref ighting, use of oxygen breathing apparatus, etc., as 
well as selected GMT and OBT types of courses while in an 
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enforced/inactive duty mode awaiting shipboard duty or trans- 
fer to another duty station. It io^ therefore, recommended 
that an effort be made to initiate a feasibility study to 
^determine the efficiency of this hypothesis by using training 
films similar to the film used in this study. 
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TABLE 1 



SUMMARY OF STUDY DESIGN 



Group 


Section 


Group 

Composition 


Film 
Viewed 


Test 
Taken 


Type 
Test 


1 Pre- 
j Test 


i Post- 
' Test 


Control #1 


A 


Recruits 


OBA Film MN- 


31169 


Pre/Post 


Pencil / Paper 
Questionnaire 


; 30 


' 30 


Control #1 


B 


Recruits 


OBA Film MN- 


3116 9 


Pre/Post 


Pencil / Paper 
Questionnai re 


30 


30 


Control #1 


C 


Recruits 

. 1 


OT^A Film MKI- 

1 


J X X u ^ 


Prc/Post 


Pencil / Paper 
Questionnai re 


30 


30 


Subtotal 


: 

J 










t 


92 


92 

j 


,Exper imen'* 
tal ^11 


f — 

U 

1 


KcC r ui ls 


OBA Fil"^ MN" 
(Modified) * 


31169 


ir r 8/ irO S X, 


r 1 imea 

Questionnaire 






Exper imen- 
tal iill 


hi 

( 


Kecrui ls 


OBA Film MM- 
(Modif ied) * 


31169 


pre/ pos L 

1 


r iimea 

Questionnaire 




0 A 
O 4 


Experimen- 
tal ^11 


F 


Recruits 


OBA Film MN- 
(Modified) * 


31169 


Pre/Post 

t 

i . 


Filmed 


« 32 


32 


vSubtotal 














100 


100 


1 

Test Total i 

No. Recruits' 

' j 


i. 

! 
1 




192 

* 


192 

1 



* Navy Film MN-31169 Modified to include pre- and post-criterion test questions. 
Subjects recorded answers to filmed questions on paper answer sheets. 
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TABLF 2 
DATA SUMMARY 
Pre-Tes t/Post-Test Means 



J JL w W 


Pre-Test 
, Mean 


Post-Test 
Mean 




rnn+--rnl T 


1 
1 










A 

XT. 


3 


.47 


10 


. 23 


1 n 
J u 


B 


3 


.53 


9 


.70 


30 


C 


3 


. 28 


10 


.75 


32 


Experimental II 












D 


3 


27 


9 


53 


34 


E 


3 


15 


9 


56 


34 


F 


3 . 


44 


9 


63 


32 
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TABLE 3 



t Test Results 
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Pre- 


Test 








Group 




N 


Mean 










Control #1 




92 


3.42 


1.79 


.33 


• 24 


.58 




IT ± ± 


1 no 

± u u 




1 47 


12 

» J- ^ 










N 


Post 
Mean 


-Test 

- ■ SD 








Control #1 




92 


10.24 


2.47 


6.18 




1.86 


Experimental 


^11 


100 


9.57 


2.50 









* t. 05=1. 98 (Ferquson, pq . 171) 
** Unbiased Estimate of Variance (Ferguson, Pg . 167) 
*** t. 05=1. 96 
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TABLE 4 



FRLQUENCY POL i HONS 




Score 



Control Group 
Experimentai Group 



16 

21 



TABLE 5 



Freq. 



22 
20 

18 
16 
14 
12 

10 

8 

6 
4 
2 




Control Group 



Lxperimentdl Group 
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TABLE 6 

SU.N TLST hiR TWO INPLl LmDENT SAMPLES* 
Median of the control group ^ experimental group observations = 9 



Control 



Experimental 



57 


35 


62 


38 



92 



100 



119 



73 



2= 192(57 X 38 - 35 x 62)^ 
^ "92 X 100 X 119 X 73 



3072 



79920400 



,0000384 



= 1 

f 

2 



X Q5 not significant 



Ferguson, pg. 355 
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TABLE 7 
GAIN/LOSS 
Pre- to Post-Test 



Pre-Test Per Cent Post-Test Per Cent % Gain/Loss 

Group N Mean (RAW) Correct Mean (Raw) Correct Pre- to Post-Test 

Control I 92 3.42 26 10.24 79 203* 

Experimental II 100 3.28 25 9.57 74 196** 



* Printed Criterion Test Questionnaire 

* Filmed Criterion Test Questionnaire 



APPENDIX A 
OBA Film MN-31169 
Criterion Test Items 
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One of the possible hazards relating to the OBA is; 



A. The handling and disposal of caustic canister chemicals. 

B. The overheating of charcoal filters. 

C. The rupturing of the oxygen pressure line. 

D. Inoperative flow meter, 

■• The OBA is designed to supply oxygen for a period of approximately: 

A. 10 to 20 minutes. 

B. 45 to 60 minutes. 

C. 90 to 120 minutes. 

D. 3 hours. 

In order to manually start the OBA system, the wearer must: 

A. Open the oxygen valve. 

B. Use own breath to fill the system. 

C. Connect the compressor line. 
Bypass the filter pack. 

. The OBA can present a hazard if the oxygen supply comes in contact 
with: 

A. Oil. 

B. Water. 

C. Foam. 

D. A hot surface. 

3. The OBA is a reliable system; however, the system will fail to functi 
if: 

A. The canister is inserted backwards. 

B. The oxygen valve is closed. 

C. The heat exchanger overheats, 

D. The flow meter is in the off position. 
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